A Hadoop Job Scheduling Algorithm Based on Pagerank
نویسندگان
چکیده
Aiming at the problem that the job scheduling algorithm based on the classical model of cloud computing in Hadoop is not high, the new job scheduling algorithm based on PageRank algorithm is proposed, Under the premise of ensuring the user experience, we propose a new job scheduling algorithm named ValidRank, which is based on the combination of hierarchical weight and waiting time. Then for the typical K Top problem and the PI problem of N, compared with the traditional job scheduling algorithm of Fair scheduler Fair and Capacity scheduler. Experimental results show that the improved ValidRank scheduling algorithm can improve the performance of the algorithm in multi-layer iterative environment.
منابع مشابه
Research on Job Scheduling Algorithm in Hadoop
On the basis of researching Fair Scheduling Strategy deeply in Hadoop cluster,the Node Health Degree is defined by constructing the relationship function between node load and job fail rate, and a job scheduling algorithm based on Node Health Degree is proposed in this paper. Nodes are grouped, according to Node Health Degree, into three categories in order to assign corresponding job in accord...
متن کاملThe Improved Job Scheduling Algorithm of Hadoop Platform
[Abstract] This paper discussed some job scheduling algorithms for Hadoop platform, and proposed a jobs scheduling optimization algorithm based on Bayes Classification viewing the shortcoming of those algorithms which are used. The proposed algorithm can be summarized as follows. In the scheduling algorithm based on Bayes Classification, the jobs in job queue will be classified into bad job and...
متن کاملHadoop Scheduling Base On Data Locality
In hadoop, the job scheduling is an independent module, users can design their own job scheduler based on their actual application requirements, thereby meet their specific business needs. Currently, hadoop has three schedulers: FIFO, computing capacity scheduling and fair scheduling policy, all of them are take task allocation strategy that considerate data locality simply. They neither suppor...
متن کاملDon't cry over spilled records: Memory elasticity of data-parallel applications and its application to cluster scheduling
Understanding the performance of data-parallel workloads when resource-constrained has significant practical importance but unfortunately has received only limited attention. This paper identifies, quantifies and demonstrates memory elasticity, an intrinsic property of dataparallel tasks. Memory elasticity allows tasks to run with significantly less memory that they would ideally want while onl...
متن کاملOptimality of the flexible job shop scheduling system based on Gravitational Search Algorithm
The Flexible Job Shop Scheduling Problem (FJSP) is one of the most general and difficult of all traditional scheduling problems. The Flexible Job Shop Problem (FJSP) is an extension of the classical job shop scheduling problem which allows an operation to be processed by any machine from a given set. The problem is to assign each operation to a machine and to order the operations on the machine...
متن کامل